This is great 👍

Will have a closer look soon.. Does the memory profile exclude recursive object references?

Does the memory profile exclude recursive object references?

Yes, Pympler only counts objects once.

A simple example:

>>> print(asized([1, 2, 1], detail=2).format())
[1, 2, 1] size=152 flat=88
    1 size=32 flat=32
    2 size=32 flat=32
    1 size=0 flat=0

A recursive example:

>>> a = [1, 2, 3]
>>> b = [4, 5, 6]
>>> a.append(a)
>>> b.append(a)
>>> print(asized(b, detail=4).format())
[4, 5, 6, [1, 2, 3, [...]]] size=432 flat=120
    [1, 2, 3, [...]] size=216 flat=120
        1 size=32 flat=32
        2 size=32 flat=32
        3 size=32 flat=32
        [1, 2, 3, [...]] size=0 flat=0  # the recursive obj has a size of 0
    4 size=32 flat=32
    5 size=32 flat=32
    6 size=32 flat=32

(note recursive objects have a size of 0 because the pointer overheads belong to the flat component of the containing data structure)

However beware, Pympler only counts memory once, but it does so the first time it encounters an object which means you can get very different results for something which is fundamentally the same measurement:

>>> a = list(range(1000))
>>> b = ['x', 'y', a]
>>> c = ['z', a]

The list `b` is huge but `c` is tiny:
>>> print(asized([b, c], detail=1).format())
[['x', 'y', [0, 1, 2, 3, 4, 5, 6, 7, 8...., 993, 994, 995, 996, 997, 998, 999]]] size=41520 flat=80
    ['x', 'y', [0, 1, 2, 3, 4, 5, 6, 7, 8,....2, 993, 994, 995, 996, 997, 998, 999]] size=41304 flat=88
    ['z', [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 1....2, 993, 994, 995, 996, 997, 998, 999]] size=136 flat=80

Wait, what, now it's the other way around:
>>> print(asized([c, b], detail=1).format())
[['z', [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, ...., 993, 994, 995, 996, 997, 998, 999]]] size=41520 flat=80
    ['z', [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 1....2, 993, 994, 995, 996, 997, 998, 999]] size=41240 flat=80
    ['x', 'y', [0, 1, 2, 3, 4, 5, 6, 7, 8,....2, 993, 994, 995, 996, 997, 998, 999]] size=200 flat=88

In Cylc the main offender for this is the config object which will be associated with whatever object references it first.

Adding the WIP label as this should to be generalised in light of the desired async main-loop endgame.

Note:

• Should have one test failure tests/database/05-lock-recover-100.t
• Unittests are not being run ATM as the modules are [purposefully] missing the [superfluous] test_ prefix.
• The plugin interface will change from before, during, after to @main_loop.before, .during, .after.

Remaining test failure turns out to be a faff, test suite behaves differently when run outside the battery.

Note: issue probably triggered by health check interval

Here is an overview of what got changed by this pull request:

Issues
======
+ Solved 2
           

Complexity increasing per file
==============================
- cylc/flow/main_loop/log_memory.py  6
- cylc/flow/main_loop/log_data_store.py  7
- cylc/flow/main_loop/__init__.py  7
- cylc/flow/main_loop/health_check.py  3
- cylc/flow/host_select.py  15
- cylc/flow/main_loop/log_main_loop.py  5
- cylc/flow/main_loop/auto_restart.py  10
- cylc/flow/parsec/OrderedDict.py  8
- cylc/flow/scripts/cylc_psutil.py  4
         

Complexity decreasing per file
==============================
+ cylc/flow/scheduler_cli.py  -1
         

See the complete overview on Codacy

Ok, I've added decorator functions for writing plugins:

from cylc.flow.main_loop import periodic

@periodic
def monitor_something(sched, state): pass

This should be forward compatible with the sort of stuff we may want to do later:

from cylc.flow.main_loop import task_event

@task_event
def monitor_task_event(task_event, state): pass

Deconfilcted.

Rebased (should remove unit test failure)...
... (it did)

@dwsutherland tagged you as a reviewer as you'd commented before, feel free to reassign if swamped by ops work, the graphql stuff is more important.

Deconflicted.

(plus some bonus documentation)

Interesting new dependency added to Cylc Flow (optional I think for some plugins): https://pypi.org/project/kiwisolver/. Not sure where the name comes from, but the authors call it Kiwi. And quite sure one of the authors is the creator of Phosphor.JS (now Lumino).

Maybe there's a way of doing it without adding to setup.cfg?
(otherwise should probably be in the docs... Thought that was the point of the above entry_points)

That's what I thought. That users would need to just install the my-plugin project via pip to be able to use in Cylc Flow, without changing Cylc Flow's files.

When do we want to merge this?

I think we can merge it once @oliver-sanders has address the question about how to add a plugin. I also thought pip install my-plugin should be enough.

It would be helpful to get a list of available plugins in the error message.

Good idea.

Done.

UserInputError: No main-loop plugin: "elephant"
    Available plugins:
        auto_restart
        health_check
        log_data_store
        log_main_loop
        log_memory

Interesting new dependency added to Cylc Flow (optional I think for some plugins):

Yes I saw that! Wonderful Aus Vs NZ banter there.

This is dragged in by matplotlib, which is an optional dependency that wouldn't get installed for most uses.

Great reviewing BTW, it's a big help.

Will check out code and re-run tests tomorrow morning. That should give time to others to have a play if they'd like too. Really good stuff coming with this PR! We need to post that somewhere like Discourse, some twitter, blog, etc. This is bringing plugins to Cylc in a way that I bet users will find useful and enjoy using.

Shouldn't cylc-flow be a dependency of cylc-doc now? (like in the setup.py)

There was already a dependency between cylc-doc and cylc-flow. This dependency isn't in the setup.py on-purpose because you never (other than perhaps CI at release time) want pip to install cylc-flow for you when building cylc docs. Less automatic but makes you think about what you are actually documenting.

2 approvals, LGTM 🎉